Neural networks are composed of a series of interconnected neuron-like processing elements (PEs). The strengths of the connections between PEs are represented by weights. Each PE stores a value known as a state, which is either specified by input data or computed from the PE's inputs and weights, using its transfer function. Typically, the transfer function is applied to the PE's net-input, the weighted sum of its inputs. Collectively, states are used to represent information in the short term. Long-term information or learning is represented by the weights. Neural networks learn from examples by modifying their weights. Once learning or training is completed, these networks can perform a variety of computational tasks.
Much of the use of neural networks is focused on feedforward networks. These networks have an architecture consisting of a layer of input nodes, a layer of output nodes, and optionally, some number of hidden layers in between. Input data is represented using the states of the input layer. The network's response to that data is represented by the states of the output layer. The feedforward nature of these networks results from the fact that, during an iteration, the computations flow from the input layer, through any hidden layers, to the output layer. This architecture allows the network to learn to map input states to output states approximating the correct response to the input. For example, if the input states represent the symptoms presented by a medical patient, the network is able to produce output states representing an estimation of the correct diagnosis for those symptoms.
One of the hindrances to wider acceptance of neural networks is the fact that they function largely as black boxes. It is often difficult to understand why a specific set of inputs produced a particular output. This difficulty is a result of the fact that the network's `knowledge` is encoded in the weights associated with a complex web of interconnections. It is desirable to find an explanation method for neural networks that is a way to explain a particular output in terms of the network inputs. For example, if a network is used to make loan decisions, it would be desirable to explain those decisions in terms of the input data describing the applicant. An explanation of this kind is required for negative loan decisions.
The black box problem is not as serious for two-layer feedforward networks, i.e., those without hidden layers. In these networks, the relationship between the inputs and outputs is straightforward. The magnitude and direction of the relationship between the states of an input PE and an output PE are given by the weight of the connection between the two PEs. Because the relationship between each input and output is fixed, these simple networks cannot capture variable relationships between input and outputs, such as non-monotonic relationships. Nor can they capture the interdependencies among inputs. That is to say, they cannot implement mappings in which the effect of some input on some output is dependent on the values of other inputs. They can learn only linear mappings, i.e., mappings where each output is proportional to a weighted sum of the inputs. Thus these networks are restricted to learning only a limited subset of the relationships that exist in the real world.
Networks with hidden layers are needed to learn nonlinear mappings, including non-monotonic relationships between inputs and outputs and interdependencies among inputs. Since there is no straightforward relationship between inputs and outputs in these networks, explanation is a difficult problem. One attempt at explanation uses sensitivity analysis. This technique involves changing the value of an input, iterating the network, and noting whether there is any meaningful change in the network's output. Using the medical domain as an example again, sensitivity analysis would involve changing one of the input symptoms and noting whether there is a change in the network's diagnostic output.
Explanation is also an issue in the field of expert systems. These systems are often compared to neural networks because the two technologies attempt to solve some of the same problems, namely: classification; prediction; and decision-making. Explanation is more straight-forward in expert systems than in neural networks, because the `knowledge` in an expert system is more explicit in that it is contained in a set of if-then rules known as a rulebase. In addition to explanation, expert systems possess other desirable capabilities not found in neural networks of the prior art. These capabilities include the ability to determine when enough input information is present to make conclusions; to provide intelligent direction to the user's information acquisition; and to calculate confidence measures to accompany decisions.
Unfortunately, expert systems lack the ability to learn from examples, which is the most appealing feature of neural networks. Although some attempts have been made to imbue expert systems with a learning capability, they still rely primarily on handcrafted rules as their source of `knowledge`. Thus, building an expert system to solve a particular problem requires finding a human expert in the problem domain, translating his knowledge into if-then rules, then debugging the rulebase.
Clearly, it is desirable to combine the learning ability of neural networks with the explanation and other capabilities of expert systems. One known attempt at such a combination involves an expert system which uses a neural network as its source of `knowledge`, and is thus able to take advantage of neural network learning. In addition to expert system capabilities such as explanation, this hybrid system includes an optional facility for converting the knowledge contained in the network into a set of rules. This system is described in the literature (see, for example, U.S. Pat. No. 4,730,259).
Unfortunately, the techniques used in this hybrid system of the prior art have several important limitations. Most significantly, they are applicable only to perceptron networks, also known as linear discriminant networks. With the possible exception of the input layer, the PEs of these networks are limited to tertiary states, i.e., there are only three possible state values, corresponding roughly to TRUE, FALSE, and unknown. The result is that perceptron networks cannot compute mappings as precisely as continuous-state networks, i.e., networks whose states are not limited to a set of discrete values. Even more important than the loss of precision is the fact that perceptron networks cannot be trained with backpropagation learning, which is the most popular training method for networks with hidden layers.
The explanation techniques of the prior art hybrid system also have limitations. For example, explanations can be generated only for conclusions since there is no method for explaining why the system is leaning toward one output or another prior to a definitive conclusion. Explanations take the form of rules, with the conditions (inputs or hidden layer states) on the left-hand side of the rule serving as an explanation for the conclusion on the right-hand side. Those conditions included in the rule make a larger positive contribution to the conclusion than those omitted. However, no precise measurement of contributions is produced for either the included or omitted conditions.
The prior art hybrid system is also limited in the way in which it directs the user's information acquisition. The system chooses a single input whose value is unknown but important and asks the user to give a value for that input. However, the system provides no information about the relative importance of the other unknown inputs.